We introduce INSTRUCTOR, a new method for computing text embeddings given task instructions: every text input is embedded together with instructions explaining the use case (e.g., task and domain descriptions). Unlike encoders from prior work that are more specialized, INSTRUCTOR is a single embedder that can generate text embeddings tailored to different downstream tasks and domains, without any further training. We first annotate instructions for 330 diverse tasks and train INSTRUCTOR on this multitask mixture with a contrastive loss. We evaluate INSTRUCTOR on 70 embedding evaluation tasks (66 of which are unseen during training), ranging from classification and information retrieval to semantic textual similarity and text generation evaluation. INSTRUCTOR, while having an order of magnitude fewer parameters than the previous best model, achieves state-of-the-art performance, with an average improvement of 3.4% compared to the previous best results on the 70 diverse datasets. Our analysis suggests that INSTRUCTOR is robust to changes in instructions, and that instruction finetuning mitigates the challenge of training a single model on diverse datasets.

Lack of factual correctness is an issue that still plagues state-of-the-art summarization systems despite their impressive progress on generating seemingly fluent summaries. In this paper, we show that factual inconsistency can be caused by irrelevant parts of the input text, which act as confounders. To that end, we leverage information-theoretic measures of causal effects to quantify the amount of confounding and precisely quantify how they affect the summarization performance. Based on insights derived from our theoretical results, we design a simple multi-task model to control such confounding by leveraging human-annotated relevant sentences when available. Crucially, we give a principled characterization of data distributions where such confounding can be large thereby necessitating the use of human annotated relevant sentences to generate factual summaries. Our approach improves faithfulness scores by 20\% over strong baselines on AnswerSumm \citep{fabbri2021answersumm}, a conversation summarization dataset where lack of faithfulness is a significant issue due to the subjective nature of the task. Our best method achieves the highest faithfulness score while also achieving state-of-the-art results on standard metrics like ROUGE and METEOR. We corroborate these improvements through human evaluation.

Existing language models (LMs) predict tokens with a softmax over a finite vocabulary, which can make it difficult to predict rare tokens or phrases. We introduce NPM, the first nonparametric masked language model that replaces this softmax with a nonparametric distribution over every phrase in a reference corpus. We show that NPM can be efficiently trained with a contrastive objective and an in-batch approximation to full corpus retrieval. Zero-shot evaluation on 9 closed-set tasks and 7 open-set tasks demonstrates that NPM outperforms significantly larger parametric models, either with or without a retrieve-and-generate approach. It is particularly better on dealing with rare patterns (word senses or facts), and predicting rare or nearly unseen words (e.g., non-Latin script). We release the model and code at github.com/facebookresearch/NPM.

Sampling diverse programs from a code language model and reranking with model likelihood is a popular method for code generation but it is prone to preferring degenerate solutions. Inspired by collaborative programming, we propose Coder-Reviewer reranking. We augment Coder language models from past work, which generate programs given language instructions, with Reviewer models, which evaluate the likelihood of the instruction given the generated programs. We perform an extensive study across six datasets with eight models from three model families. Experimental results show that Coder-Reviewer reranking leads to consistent and significant improvement (up to 17% absolute accuracy gain) over reranking with the Coder model only. When combined with executability filtering, Coder-Reviewer reranking can often outperform the minimum Bayes risk method. Coder-Reviewer reranking is easy to implement by prompting, can generalize to different programming languages, and works well with off-the-shelf hyperparameters.

We study the problem of retrieval with instructions, where users of a retrieval system explicitly describe their intent along with their queries. We aim to develop a general-purpose task-aware retrieval system using multi-task instruction tuning, which can follow human-written instructions to find the best documents for a given query. We introduce the first large-scale collection of approximately 40 retrieval datasets with instructions, BERRI, and present TART, a multi-task retrieval system trained on BERRI with instructions. TART shows strong capabilities to adapt to a new retrieval task via instructions and advances the state of the art on two zero-shot retrieval benchmarks, BEIR and LOTTE, outperforming models up to three times larger. We further introduce a new evaluation setup, X^2-Retrieval to better reflect real-world scenarios, where diverse domains and tasks are pooled and a system needs to find documents aligning users' intents. In this setup, TART significantly outperforms competitive baselines, further demonstrating the effectiveness of guiding retrieval with instructions.

我们提出了一项实证研究，以适应现有的经过验证的文本对文本模型，以备长期输入。通过沿预训练管道的三个轴的全面研究 - 模型架构，优化目标和训练式语料库，我们提出了一种有效的食谱，以从现有的短篇小说模型中构建长篇小说模型。具体而言，我们用汇总仪的块关注替换了变压器中的全部注意力，并使用蒙版的跨度预测任务为模型预算，长度不同。就训练训练的语料库而言，我们发现，与使用通常在其域覆盖范围中通常受到限制的现有长文档语料库相比，使用大型开放域语料库的随机串联的短篇小说可以提高性能。通过这些发现，我们建立了一个长篇文本模型，该模型可以在长篇文本质量检查任务上实现竞争性能，并在五个长文本摘要数据集上建立新的最新技术，通常优于先前的方法，具有较大的模型大小。

为了解决现实世界应用需求的日益增长，知识密集型NLP（KI-NLP）的研究应通过捕获真正开放域环境的挑战：网络规模知识，结构缺乏，质量不一致，和噪音。为此，我们提出了一种新的设置，用于评估现有的KI-NLP任务，其中我们将背景语料库概括为通用Web快照。我们重新保证Kilt，最初为维基百科最初开发的标准Ki-NLP基准测试，并要求系统使用CCNet的子集 - 球体语料库 - 作为知识源。与维基百科相比，球体是较大的数量级，更好地反映了互联网上的全部知识。我们发现，尽管潜在的覆盖范围，规模挑战，结构缺乏，质量较低，来自领域的检索可以实现最先进的检索系统，以匹配和甚至优于基于Wikipedia的模型在几个kilt上任务 - 即使我们积极过滤看起来像维基百科的内容。我们还观察到Wikipedia的单一密集通道指数可以胜过稀疏的BM25版本，而在球体上尚不实现。为了促进进一步研究该领域，并尽量减少社区对专有黑匣子搜索引擎的依赖，我们将分享我们的指数，评估指标和基础设施。

我们提出了Drboost，一个受升压启发的密集检索合奏。Drboost在阶段接受培训：通过仅关注当前合奏制作的检索错误来依次学习和专注于每个组件模型。最终的表示是所有组件模型的输出矢量的串联，使其成为测试时间标准密集检索器的替代品。与标准密集检索模型相比，Drboost享有几个优点。它产生的表示是4x更紧凑，同时提供可比的检索结果。它还在具有粗量化的近似搜索下进行令人惊讶的良好，从而减少另一个4x的延迟和带宽需求。在实践中，这可以在从内存中服务索引之间的服务指数之间的区别，为更便宜的部署铺平道路。

许多NLP任务需要处理超出预磨模模型的长度限制的长语境。为了将这些模型扩展到更长的文本序列，已经提出了许多有效的远程注意力变体。尽管沿着这个方向进行了丰富的研究，但仍然难以在实际用例中衡量这些模型的相对有效性，例如，如果我们在预先rain-yfetune范式之后应用这些模型。在这项工作中，我们的目标是对这些具有大规模和受控实验的这些新兴模型进行彻底的分析。对于每个关注变体，我们使用相同的长DOC语料库，然后使用相同的长DOC语料库，然后为现实世界的长情节任务进行芬特这些模型。我们的调查结果揭示了现有广泛使用的远程基准的陷阱，并显示任何经过测试的高效关注可以在标准预介质范式下击败一个简单的本地窗口关注。对本地注意力变化的进一步分析表明，即使是常用的注意力窗口重叠也没有必要实现良好的下游结果 - 使用不相交的本地关注，我们能够构建符合性能的更简单且更高效的Long-Doc QA模型霍尔福勒〜\ citep {longformer}其预先花费的一半。

最近的参数效率语言模型调整（PELT）方法可以使微调的性能与较少的可训练参数相匹配，并且在训练数据受到限制时尤其表现良好。但是，不同的PELT方法在相同的任务上的性能可能会有所不同，因此为特定任务选择最合适的方法是不平凡的，尤其是考虑到快速增长的新PELT方法和任务。鉴于模型多样性和模型选择的难度，我们提出了一个统一的框架Unipelt，该框架将不同的毛皮方法纳入了子模型，并学会了激活最适合当前数据或通过门控机制设置的方法。在胶水基准上，与最佳的单个毛皮方法相比，UniPelt始终达到1〜4％的增长，而其融合甚至超过了不同设置下的微调。此外，UniPelt通常超过上限，该上限在每个任务上单独使用的所有子模型的最佳性能，表明多种PELT方法的混合物可能本质上比单个方法更有效。